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ABSTRACT 


Blended, flipped, and semi-flipped instructional approaches were used in various sections of a 
numerical methods course for undergraduate mechanical engineers. During the spring of 2014, a 
blended approach was used; in the summer of 2014, a combination of blended and flipped instruc¬ 
tion was used to deliver a semi-flipped course; and in the fall of 2014, a fully-flipped approach was 
taken. Blended instruction aims to integrate technology-driven instruction with face-to-face learning 
and is often used to enhance the traditional lecture. With “flipped” instruction, students practice 
skills during class after viewing or/and reading lecture content beforehand. To directly assess these 
instructional methods, we compared multiple-choice and free response results from identical final 
exams. We did this for all students as well as demographic segments of interest to our research, 
including underrepresented minorities and transfer students. We uncovered several differences 
having medium to large effect sizes, suggesting that some degree of flipped instruction may have 
been more beneficial than blended learning for both lower and higher-order skills development. The 
students rated the classroom environment using Fraser’s College and University Classroom Envi¬ 
ronment Inventory (CUCEI). The three classroom environments were statistically similar with small 
effect sizes. Flowever, there was a trend in lower ratings for the flipped and semi-flipped classrooms 
versus the blended classroom across the various environmental dimensions. This may indicate that 
blended instruction had the most desirable classroom environment. Based on an evaluation survey, 
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only 38% of respondents preferred flipped instruction to usual methods, although 54% preferred 
active learning to lecture. In an open-ended question, the most frequently-stated benefits of flipped 
instruction involved enhanced learning or learning processes, and engagement and professional 
behaviors. These results aligned with our focus group results. This study is believed to be one of 
the first to compare these three modalities in a STEM course. 

Key words: Flipped class, Blended instruction, Numerical Methods, Mechanical Engineering. 


INTRODUCTION 


The teaching of STEM courses solely through the traditional lecture is emerging as an ineffective 
and inferior method of instruction (Mazur, 2009; Freeman et al., 2014; Wieman, 2014). When topics 
are complex, students require in-depth engagement activities to practice their higher order think¬ 
ing skills and fully construct an understanding of the topic (Garrison & Vaughan, 2008). Blended 
learning has emerged in higher education as a means to provide more engaging, quality-driven 
experiences for learners. It aims to optimally integrate face-to-face teaching with online learning 
(Garrison & Vaughan, 2008; Bourne et. al, 2005; Dziuban et al., 2006). With blended learning, as¬ 
pects of face-to-face classroom learning are replaced or enhanced by online or technology-based 
experiences, such as simulations, labs, tutorials, and assessments (Garrison & Vaughan, 2008). 
Those who employ blended learning have the objective of “using the web for what it does best, 
and using class time for what it does best” (Osguthorpe & Graham, 2003, 227). Thus, blended 
learning is the convergence of two historically separated teaching models - traditional face-to- 
face and more recent computer-mediated models, which often aim to accommodate collabora¬ 
tive human interaction in the form of virtual communities (Graham, 2006). A related approach, 
the flipped classroom, has also emerged in higher education. This approach frees class time for 
higher-engagement activities such as problem solving. This is typically accomplished by having 
students watch online videos with lecture content before class (Bergmann & Sams, 2012). Thus, 
with “flipped” instruction, students apply skills during class after preparing with lecture content 
beforehand, and instructors serve in a support role as guide and coach during this skills applica¬ 
tion (Velegol et al., 2015). 

Studies have shown that active or interactive learners achieve significantly better results and gains 
compared to passive learners in problem solving, time to mastery, and conceptual understanding 
(Chi, 2009; Hake, 2001). A recent meta-analysis of studies comparing active learning to traditional 
lecturing indicated that student test performance increased by about half a standard deviation on 


2 


FALL 2016 





ADVANCES IN ENGINEERING EDUCATION 

Comparing the Effectiveness of Blended, Semi-Flipped, and Flipped Formats 
in an Engineering Numerical Methods Course 



average in active-learning STEM courses. Also, average failure rates were 22% with active learning 
compared to 34% with traditional lecturing (Freeman et at, 2014). In addition, educators have 
stressed that genuine learning occurs when students discuss, analyze, apply, problem solve, design, 
and otherwise perform active learning (Prince, 2004; Bonwell & Eison, 1991). 

An NSF grant entitled “Improving and Assessing Student Learning in an Inverted STEM Class¬ 
room Setting” allowed us to compare active learning methods, specifically blended, flipped, and 
semi-flipped instruction, in a numerical methods course at the University of South Florida (USF) 
over the course of three semesters (Kaw et.al., 2013). This required class is taken by junior and se¬ 
nior mechanical engineering students and covers numerical methods for differentiation, nonlinear 
equations, simultaneous linear equations, interpolation, regression, integration, and ordinary dif¬ 
ferential equations. We began a formal comparison of blended instruction vs. flipped instruction vs. 
a combination of the two in the 2014 spring semester. This is one of the first studies we are aware 
of that compares these three methods in a STEM course. 

The spring 2014 course was taught in a blended fashion and consisted of in-class lectures, clicker 
questions, and exercises. In addition, there was an online discussion board, lecture videos, and online 
quizzes. This course aligned with the supplemental model of blended instruction, which retains the 
structure of the traditional class but adds technology-based activities outside of class to enhance 
student engagement and offer additional resources (Twigg, 2003). The instructor had extensive prior 
experience in teaching this course in a blended manner. “Mixing it up” in the summer term, half of 
the topics were taught in a blended mode and the other half in a flipped mode. In the flipped ses¬ 
sions, students watched video lectures and/or read the textbook before class. During class, students 
worked in teams on exercises or applications and used clickers. Micro-lectures were employed as 
necessary, and the online discussion board was also available. In the fall term, the course was fully 
flipped using these same resources. 

In investigating and comparing these instructional methods, our research questions were as 
follows: 

1. Is there a difference in numerical methods achievement level, in particular for various student 
demographic groups, when using blended, flipped, and semi-flipped instructional approaches? 

2. Are there differences in students’ perceptions of the learning environment within blended, 
flipped, and semi-flipped numerical methods classrooms? 

3. What are the benefits and drawbacks of flipped instruction in numerical methods based upon 
the students’ perspectives? 

By investigating these research questions, we aim to develop recommended practices for teaching 
numerical methods using technology-enhanced and/or active-learning approaches. Our objective 
is to inform faculty who are interested in using these types of teaching and learning approaches 


FALL 2016 


3 






ADVANCES IN ENGINEERING EDUCATION 

Comparing the Effectiveness of Blended, Semi-Flipped, and Flipped Formats 

in an Engineering Numerical Methods Course 


for numerical methods and other engineering courses. The lessons learned have the potential to 
change the way numerical methods and other engineering courses are taught. 

Therefore, in the following sections, we further review the existing literature on blended and flipped 
classrooms, including for mechanical engineering courses and courses directly related to numerical 
methods. We then discuss our methods, including course delivery using each of the three modalities 
as well as our data collection and statistical analysis methods. This is followed by a detailed com¬ 
parison of the multiple-choice and free-response exam results for these instructional approaches, 
including for various demographic segments of interest as well as for various numerical-methods 
topic areas. We also provide a comparison of the three methods in terms of the classroom environ¬ 
ment, and we present students’ evaluations and perceptions of the benefits and drawbacks of the 
flipped classroom as gathered through a survey and focus group. 


LITERATURE REVIEW 


Blended and Flipped Classrooms: Descriptions and Motivations from the Literature 

A major initiative with blended learning was an instructional redesign program that encouraged 
higher educational institutions to redesign their instruction using technology. Supported by the Pew 
Charitable Trusts, some of the improvement techniques included computer-based assessments and 
feedback, online student discussion groups and learning communities, computer-lab group work 
with faculty present in lieu of a lecture, and online interactive tutorials with assessment checks 
(Twigg, 2003). The University of Central Florida (UCF) has been a large adopter of the blended 
learning model and is a recognized leader in this area, having offered blended courses since 1997 
(Dziuban et al., 2006; Cavanagh, 2011). The goals of blended learning have been identified as fol¬ 
lows: pedagogical richness (to include enhanced or deeper student learning), student access to 
knowledge, social interaction in-class and online, student self-directedness and learning choices, 
cost effectiveness, and flexible and responsive learning atmospheres (Osguthorpe & Graham, 2003). 
There is evidence that blended learning can be more effective than a traditional instructional ap¬ 
proach (Garrison & Kanuka, 2004). 

Blended learning has been advocated or implemented in other mechanical engineering courses 
besides numerical methods, in which online experiments, web-based simulations, or remote labs 
have been used (Cortizo et al., 2010; Restivoet al., 2009; Flenning et al., 2007; Hu & Zhang, 2010). 
Blended learning has also been used in courses that are foundational to numerical methods, such 
as introductory computer programming. In a large, first-year introductory computing course for 
engineers, an online, self-practice tool was put in place to provide an additional, flexible learning 
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resource for students, in which they could assess their understanding, receive automatic feedback, 
and subsequently seek one-on-one assistance from instructors as needed (El-Zein et al., 2009). 

Despite his long-time use of blended instruction in this course, the instructor developed an 
interest in flipping the course. The flipped classroom enables face-to-face time for application of 
skills with the instructor present for support. The instructor’s primary goals were to promote higher 
order thinking and metacognitive skills. In addition, he wanted to drive student responsibility for 
learning, albeit using fully guided instruction (Clark et al., 2012). Similarly, in a very recent survey 
taken by almost 1,100 faculty members primarily from the US and Canada, the top motivations for 
flipped classrooms included increasing student engagement (79%) and improving student learning 
(76%) (Bart, 2015). In a second recent survey by the National Center for Case Study Teaching in 
Science, case-study teachers indicated that they teach in a flipped manner to increase interaction 
with students, promote flexibility, and increase student involvement in and engagement with their 
learning (Herreid & Schiller, 2013). This is in agreement with other sources that describe flipped 
instruction as increasing student-to-teacher interaction and student collaboration as well as 
providing self-paced “pause and rewind” capability and other benefits (Bergmann & Sams, 2012; 
Rosenberg, 2013). 

In the discipline of numerical methods, a controlled study of the flipped classroom for a numeri¬ 
cal methods course for engineers was recently done at Utah State University. A treatment section 
(i.e., flipped) as well as a comparison section (i.e., traditional lecture) were run concurrently (Bishop 
& Verleger, 2013; Bishop, 2013). The flipped section made use of multiple resources developed as 
part of the Holistic Numerical Methods initiative (HNM, 2015), shown in Figure 1, in addition to other 
previous work by Kaw and colleagues, including video lectures, and a concept test (Kaw & Yalcin, 
2012; Kaw et al., 2012). Kaw and Hess previously investigated the effectiveness of the following four 
instructional delivery modalities for a single numerical methods topic area (i.e., nonlinear equations): 
1) traditional lecture, 2) web-enhanced lecture, 3) web-based self-study, and 4) web-based self-study 
and classroom discussion (i.e., flipped mode). The last modality was found to be best for student 
performance, and the second was best for student satisfaction (Kaw & Hess, 2007). 

The flipped classroom has been implemented with other mechanical engineering courses as well, 
including introductory mechanical design, statics and mechanics, and electronics instrumentation, 
as discussed in the literature (Dollar & Steif, 2009; Steif & Dollar, 2012; Cavalli et al., 2014; Connor et 
al., 2014; Papadopoulos & Roman, 2010). Online statics materials developed as part of the Carnegie 
Mellon Open Learning Initiative (OLI) have been used in both a flipped and blended fashion (Dollar 
& Steif, 2009; Steif & Dollar, 2012; Steif & Dollar, 2009). The flipped classroom is also being imple¬ 
mented in math courses such as calculus and linear algebra, which provide foundational knowledge 
for numerical methods (McGivney-Burelle & Xue, 2013; Talbert, 2014; Love et al., 2014). 
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Figure 1. Holistic Numerical Methods Courseware (http://nm.MathForCollege.com. ) 


Blended and Flipped Classrooms: Results from the Literature 

In comparisons of blended versus traditional learning, blended learning has shown success. In 
the first round of the Pew Trusts’ instructional redesign projects, which involved a range of arts 
and sciences courses, five of the ten projects reported improved learning outcomes. Four reported 
equivalent achievement, and one was inconclusive. The improved learning outcomes included sig¬ 
nificantly higher grades and exam performances. In addition, seven of the ten redesigns showed an 
improvement in course completion and retention rates, including decreased D/F/drop/withdrawal 
rates (Twigg, 2003). Assessment of face-to-face vs. fully online vs. blended courses at UCF has 
shown blended learning to have the highest level of student success, as defined by the percent of 
students earning a C or better. Blended courses also had the lowest withdrawal rate and the largest 
percentage of “excellent” ratings by students on the course evaluations (Cavanagh, 2011). 

In mechanical and other engineering courses, student perceptions of blended learning have been 
favorable also. In a laboratory course that used a remote experiment for measuring mechanical 
properties, the students evaluated the impact of the remote lab. On a scale of 1 to 7, the students 
rated “deeper learning of previous knowledge” with an average value of 5.6 and “e-learning con¬ 
tribution for better learning quality” at 5.7 (Restivo et al., 2009). In the introductory programming 
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course discussed previously, 75% of student respondents found the self-practice tool to be useful or 
very useful. In addition, the overall student satisfaction with the quality of the course rose 23% after 
implementation of the online tool (El-Zein et at, 2009). A group of 21 graduate engineering courses 
was taught in a traditional fashion and subsequently in a blended fashion to measure differences 
between the methods in terms of student satisfaction, class attendance, motivation, and collabora¬ 
tion. Survey responses from approximately 800 students were collected, and students perceived 
statistically greater levels of satisfaction, motivation, and collaboration with their classmates in the 
blended environment. Students also reported statistically higher attendance rates in the blended 
classrooms and identified access to their instructor as the strongest predictor of satisfaction with 
blended learning (Martinez-Caro & Campuzano-Bolarin, 2011). 

With flipped instruction, comparisons to traditional instruction for other mechanical engineering 
courses have shown mixed results, both in terms of direct and indirect assessments. For example, at 
the University of Puerto Rico, an inverted classroom was implemented for several sections of a stat¬ 
ics course (Papadopoulos & Roman, 2010). On an end-of-course Concept Assessment Tool in Statics 
(CATS), students in the inverted sections scored statistically higher than students in the traditional 
sections (p= 0.0076). At the University of North Dakota (UND), where a series of undergraduate me¬ 
chanical engineering courses was flipped, students in a traditional section of a numerical methods 
course exhibited a higher level of achievement compared to students in a flipped section. In the tradi¬ 
tional section, 82% earned a C or better, compared to 72% in the flipped section (Cavalli et at, 2014). 
The controlled study for the numerical methods course at Utah State examined students’ problem 
solving on usual assessments (i.e., homework and exams) as well as their understanding via a concept 
test developed by Kaw and colleagues (Kaw & Yalcin, 2012) and concept-based quizzes. The exam 
and quiz averages and the concept test gains were statistically equivalent between the treatment and 
comparison sections. Although average homework scores were statistically lower for the treatment 
section, it was believed that these students had a larger workload and had to prioritize other tasks over 
the homework assignments, which were worth a relatively small portion of the grade (Bishop, 2013). 

In courses related to numerical methods or other engineering courses, the results have also been 
mixed. In a linear algebra course, students in the flipped section performed similarly to students in 
the traditional section on the final exam focused on conceptual understanding (Love et al., 2014). 
However, in a calculus course, students who received flipped instruction on certain topics, versus 
those who did not, scored four to five points higher on an exam and assignments that covered the 
topics (McGivney-Burelle & Xue, 2013). We did not uncover articles in our literature search discuss¬ 
ing an approach similar to our semi-flipped approach, in which half of the course had been flipped 
and the other half had been run in blended mode. Rather, we uncovered studies such as the calculus 
study, in which one or a few units had been flipped within an otherwise traditionally-run course. In 
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an environmental engineering course at Penn State, there was no statistical difference in the final 
exam scores across six semesters in which the course had been taught both traditionally and in 
flipped mode (Velegol et al., 2015). Interestingly, in the recent faculty survey discussed previously, 
although 75% of faculty saw greater student engagement with the flipped classroom, only one-half 
(55%) noted evidence of improved student learning (Bart, 2015). 

In terms of student perceptions of and preferences for the flipped classroom, results have also been 
mixed, as noted previously (Bishop & Verleger, 2013). Fifty-four percent (54%) of the UND learners in 
the flipped sections preferred the flipped format (Cavalli et al., 2014). As will be discussed in the results 
section, this was higher than our percentage of students who preferred the flipped format for numeri¬ 
cal methods. Similarly, in a flipped electronics instrumentation course taken by mechanical engineers 
at Rensselaer, 56% indicated a preference for online video lectures versus traditional lectures (Connor, 
2014). In a comparison of student opinions of the learning experience in the Utah State study, the ratings 
were statistically equivalent between the treatment and comparison sections except for three items on 
the 18-item survey. These three items included learning to apply course material and an overall rating of 
the course. They were rated statistically lower by the treatment section and had medium effect sizes. 


METHODS 


To assess and compare the outcomes of the three methods for delivering numerical methods 
instruction, we developed a comprehensive plan consisting of direct and indirect assessments. We 
used final exam scores to directly assess and compare achievement with blended versus semi- 
flipped versus flipped instruction in numerical methods. The assessment analyst also conducted 
interviews and discussions with the instructor to uncover gains that may not have been apparent 
in the exam data. A student demographics survey was administered so the exam results could be 
analyzed for specific segments of the population, such as females, underrepresented minority stu¬ 
dents, or transfers to the college of engineering. In addition, the students were indirectly assessed 
for their perceptions using classroom environment and flipped classroom evaluation surveys. Finally, 
student focus groups were conducted with different demographic segments to assess benefits and 
drawbacks of flipped instruction. We will first discuss the methods used to develop and deliver the 
course and the participants in the three modes. 

Course Delivery Methods and Student Participants 

In the blended version of the course, students were assigned to view primers for the pre-requisite 
course material and take an auto-graded, online quiz prior to class. For the topics covered in class, 
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an auto-graded, online quiz was assigned after class and due four hours before the next class. These 
quizzes consisted of about 90% algorithmic questions and 10% multiple-choice questions and were 
administered through the university learning management system, counting for 15% of the student’s 
grade. The algorithmic feature enabled the same question to be presented to all students, but with 
random numerical parameters, thereby reducing opportunities for academic dishonesty. During class, 
lecture occurred, and clickers were used frequently (i.e., in approximately half of the class sessions) 
with peer interaction followed by instructor discussion. The clickers were used formatively, and a 
grade was not assigned to the responses. The students also worked on short exercises, assisting 
one another as needed. Some of these exercises were assigned as homework to be completed for 
a grade. The Piazza online discussion board was available continuously (i.e., 24/7) for very quick 
feedback from the instructor, TA and fellow students, as were the video lectures (Piazza, 2015). A 
summary of the teaching methods is given in Table 1. 


Activity 


Blended 


Flipped 


Pre-class Study pre-requisite material via videos for one-half of 
the course topics. 

24/7 access to open courseware & Piazza discussion 
board. 


In-class Clicker quiz in half of class sessions to gauge 
conceptual understanding (not graded). Fewer 
questions presented vs. in flipped class. Questions 
answered correctly by fewer than 75% are discussed 
peer-to-peer followed by re-polling. 

Mostly lecture with active learning components (e.g., 
two-way questioning, clickers, short exercises with 
peer interaction); some graded. 

Post-class Automatically-graded quizzes (due before next class). 

Problem set of ~6 questions; not graded. 

2-3 graded programming projects analyzing 
experimental data. 

Some in-class exercises assigned as homework; some 
graded. 


Study topic via textbook or video lectures. 


24/7 access to open courseware & Piazza discussion 
board. 

Automatically graded quiz (due 3 hours before class). 

Essay question on most difficult or interesting 
concept from videos (due 3 hours before class). 

Clicker quiz in every class session to gauge 
conceptual understanding (not graded). Questions 
answered correctly by fewer than 75% are discussed 
peer-to-peer followed by re-polling. 

Micro-lectures based on pre-class quiz and responses 
to essay question. 

Short exercises or outline-the-solution problems with 
peer interaction and instructor help; some graded. 

Automatically-graded quizzes (due before next class). 

Problem set of ~6 questions; not graded. 

2-3 graded programming projects analyzing 
experimental data. 


Note: In the semi-flipped class, approximately half of the class sessions were blended, and the others were flipped. 

Table 1. Comparison of Blended and Flipped Delivery Methods. 
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In the flipped version of the course, students were assigned to watch lecture videos and/or study 
the textbook before class. In addition, an auto-graded online quiz and an open-ended question were 
due four hours before class. The open-ended question prompted the student for the most difficult 
or interesting topics encountered in the videos. These were read by the instructor prior to class, 
and most of them were addressed during class through micro-lectures. Others were addressed 
via the Piazza discussion board, using established links and blogs, text, and PDF files with worked 
examples. Clickers were used almost daily. Students completed exercises or outlined solutions to 
an applied problem with their peers. Auto-graded, online quizzes were assigned after class on the 
topics covered in class. The quizzes again counted for 15% of the student’s grade. A presentation 
showcasing how the flipped class was taught can be accessed at http://www.eng.usf.edu/-kaw/ 
flipped/autar_kaw_flipped_classroom_workshop_2015.pptx. 

In the semi-flipped version of the course, a portion (i.e., approximately half) of the class sessions 
were conducted as blended classes, while the others were conducted as flipped classes. Four of 
the eight topics were delivered in the flipped mode, and the other four were taught in the blended 
fashion. The choice of the topics for each instructional mode was based on students’ final exam 
performance over several previous semesters. Also during this semester, eight of the 14 multiple- 
choice questions on the final exam were based on material taught in the flipped mode, and the other 
six were based on material taught in a blended manner. 

The video lectures used for all three instructional methods were developed as part of previous 
NSF-funded work. This open courseware, known as Holistic Numerical Methods (HNM), includes 
resources for introductory computing, differentiation, nonlinear equations, simultaneous linear 
equations, interpolation, regression, integration, ordinary differential equations, partial differential 
equations, optimization, and Fast Fourier transforms (Kaw et al., 2012). Between 2009 and 2012, 
250 videos were developed and recorded for this open courseware initiative, with an average length 
of about nine minutes. The videos can be accessed using the following link: http://mathforcolleqe. 
com/nm/videos/i ndex.html . 

Study Participants 

The following numbers of students were enrolled in the blended, semi-flipped, and flipped courses: 
95, 27, and 58 respectively, totaling 180 students across three semesters from the spring 2014 to the 
fall 2014. The numbers of participants who provided data for this study were as follows: 73 (77%), 
18 (67%), and 41 (71%), respectively, totaling 132 students. Of the 132 students, 18 (14%) were female. 
Additional demographic characteristics of the participants can be determined based on the sample 
sizes shown in Table 3 in the results section. 
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Assessment of Learning 

Direct assessment of student learning based upon the final exam was used to investigate 
our first research question comparing achievement with the three methods. The final exam 
contained 14 multiple-choice questions that were identical across the three semesters. The 
multipie-choice questions were designed to test the lower-level skills in Bloom’s taxonomy (Wig¬ 
gins & McTighe, 2005). In addition, there were four open-ended, free-response questions that 
remained the same from term to term. These were intended to measure the higher-order skills 
in Bloom’s taxonomy, such as synthesis. Using the multiple-choice and free response data, we 
compared the three instructional methods using an analysis of covariance (ANCOVA), with the 
pre-requisite GPA as the covariate or control variable. Since the summer term sample size was 
less than 20, we also ran a non-parametric analysis of covariance, known as Quade’s test (Quade, 
1967; Lawson, 1983). The p-values based on the parametric and non-parametric analyses were 
generally in agreement, and examining both served to corroborate the results. Nonetheless, 
we defaulted to the non-parametric result given the small sample sizes. We used Tamhane’s T2 
post-hoc comparison, which is conservative, to test significance between the individual pairs 
(Field, 2005). These analyses were conducted using the software SPSS 21. The pre-requisite GPA 
was based on the grades received in Calculus 1-3, ordinary differential equations, introductory 
programming, and physics 1. 

We conducted a series of ANCOVA’s and Quade’s tests to analyze the data in a stratified fashion, 
comparing the instructional methods for each demographic segment of interest to our research. 
For example, we were interested in questions of the type: “For females, which instructional method 
is associated with the highest achievement?” We also ran an analysis of variance (ANOVA), its 
non-parametric variant (i.e., the Kruskal-Wallis test), and Tamhane post-hoc tests to determine if 
the instructional method was a factor for students of different achievement levels, as determined 
by the pre-requisite GPA. We used three GPA groups - under 3.00, between 3.00 and 3.49, and 
3.50 and above. Finally, we ran an ANCOVA for each of the seven numerical methods topic areas 
to determine if the class as a whole performed better with flipped versus blended instruction for 
particular numerical methods topic areas. 

Given the large number of tests we ran for each analysis family, we applied the Bonferroni cor¬ 
rection to the individual p-values (Perneger, 1998; Bland & Altman, 1995). When a large number of 
statistical tests are performed, some will unfortunately result in p < 0.05 just by chance; therefore, 
if one continues to test long enough, a significant result will eventually occur (McDonald, 2014). The 
Bonferroni correction addresses this by reducing the alpha level applied to each individual test so 
that the family-wide error rate remains at a=0.05. With this correction, the alpha level for each of our 
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individual tests was set at (0.05/m), where m is the number of tests. Unfortunately, this correction 
has the drawback that it becomes very conservative as the number of tests increases; also, the in¬ 
terpretation of a finding is dependent on the number of other tests conducted. In other words, the 
type II error rate increases such that true differences may be deemed non-significant (Perneger, 
1998). These disadvantages as well as the lack of a well-formulated definition of an “analysis fam¬ 
ily” surround the use of the Bonferroni correction (Miller, 1981). We present this information so the 
reader will be fully informed when interpreting our results. 

In addition to determining significance levels, we calculated effect sizes based on Cohen’s d 
for all pairs of means (Sullivan & Feinn, 2012; Kotrlik et al., 2011). The effect size represents the 
magnitude of the difference between two groups and provides a measure of practical or substan¬ 
tive significance. As discussed in the above two articles, both the significance level and the effect 
size should be determined and reported in order to show the overall result or the “big picture.” 
Further, the Publication Manual of the American Psychological Association advises authors to 
include both the p value and the size of the effect (American Psychological Association, 2010). 
For, in order to appreciate the importance of the findings and convey the most complete meaning 
of the results, it is necessary to provide a measure of effect size, such as Cohen's d (American 
Psychological Association, 2010). A reference for engineers also recommends interpretation of 
both the magnitude of the observed change for practical significance and the p value (Hogg & 
Ledolter, 1992). We used the following threshold values to determine small, medium, and large 
effects, as delineated by Cohen: d- 0.20 (small), d= 0.50 (medium), and d= 0.80 (large) (Cohen, 
1987; Salkind, 2010). For the ANCOVA results, we calculated adjusted effect sizes based on the 
adjusted means (Huck, 2012). SPSS adjusts the means using the mean value of the covariate 
(Norusis, 2005). 

Collection of Demographic Data for Stratified Direct Assessment 

To directly assess learning for various demographic segments in a stratified manner, we developed 
a demographics survey for this research, to be used in conjunction with the final exam. It consisted 
of questions regarding age, gender, race/ethnicity, Pell grant status, transfer status, work and credit 
hours, and pre-requisite course grades. The pre-requisite courses consisted of calculus 1-3, ordinary 
differential equations, introductory programming, and physics 1. We also accounted for the possible 
fulfillment of these requirements through Advanced Placement (AP) test credits. These grades were 
used to calculate a numerical methods pre-requisite GPA for each student to be used as a covariate 
or control variable in the analysis of covariance of the blended vs. flipped vs. semi-flipped approach. 
The students were asked to provide a personal code when completing this survey, which allowed 
us to match the student’s exam performance with his/her demographic characteristics and thereby 
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evaluate various demographic segments of our population. The demographic segments of interest 
within our research were the following: 

1. Age Group: {22 or under, over 22} 

2. Gender: {male, female} 

3. Under-Represented Minority (URM): {yes, no} 

4. Transfer Status: {admitted to engineering as a freshmen, transferred to engineering from a 
community college with an Associate’s degree, other transfer students} 

5. Pell Grant Recipient: {yes, no} 

6. Combined Work and Credit Hours Effort: {under 40, 40-65, over 65} 

The age categories reflect our interest in traditional vs. non-traditional engineering students, with 
the traditional student starting college at age 18. The under-represented minority students consisted 
of Hispanic, American Indian, Black/African American, or Hawaiian/Pacific Islander students. The 
“other” transfer students consisted of internal-to-USF transfers to the engineering school, community 
college transfers without Associates’ degrees, and transfers from external four-year programs. The 
Pell Grant Program provides need-based grants to low-income undergraduates for postsecondary 
education, and the amount depends in part on the financial need and the costs to attend school 
(Federal Pell Grant Program, 2015). The work and credit hours were combined by multiplying the 
credit hours by three and adding the weekly work hours. For the summer term, we used a slightly 
different multiplier. Sixty-five combined hours (65) corresponded to a full-time student taking 
15 credits and also handling a 20-hour-per-week work study or other job, which would not be un¬ 
usual for a student. The boundaries associated with this field resulted in a reasonable number of 
students in each category. 

Classroom Environment Survey 

We employed the College and University Classroom Environment Inventory (CUCEI) to investigate 
our second research question concerning student perceptions of the classroom environment with the 
three instructional methods (Fraser & Treagust, 1986). This reliable inventory evaluates perceptions of 
seven psychosocial dimensions of the classroom and has been used previously in flipped classroom 
research (Strayer, 2012; Clark et al., 2014). Several of the dimensions are particularly relevant to the 
flipped classroom, including student cohesiveness, individualization, innovation, involvement, and 
personalization, as shown in Table 2. There are seven questions per dimension, and each question 
has a scale of 1 to 5, with 5 being most desirable. An average score for the dimension was calculated 
for each student. These scores were then used to test for differences in the instructional methods 
by dimension. Specifically, we ran an ANOVA, Tamhane post-hoc tests, and the Kruskal-Wallis test 
for each dimension to determine if the instructional method was a significant factor in students’ 
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Dimension 

Definition 

Student Cohesiveness 

Individualization 

Innovation 

Involvement 

Personalization 

Satisfaction 

Task Orientation 

Students know & help one another 

Students can make decisions; treated individually or differentially 

New or unusual class activities or techniques 

Students participate actively in class 

Student interaction w/ instructor 

Enjoyment of classes 

Organization of class activities 

Table 2. CUCEI Dimensions. 


perceptions of the classroom environment. We distributed the CUCEI during the last week of class 
and collected the data anonymously to enable the most comprehensive and honest viewpoints. The 
students were offered extra credit for completing the surveys. 

Flipped Classroom Evaluation Survey and Student Focus Groups 

A flipped classroom evaluation survey and student focus groups were used to investigate our third 
research question about the benefits and drawbacks of flipped instruction in numerical methods 
from the students’ perspectives. Our evaluation survey was modeled upon a previously-developed 
survey and used for both formative and summative assessment purposes. We employed many of 
the questions used by Zappe, Leicht and colleagues at Penn State, who used perception instru¬ 
ments in a flipped architectural engineering course (Zappe et al., 2009; Leicht et al., 2012). We used 
questions pertaining to preferences for flipped instruction and in-class active learning as well as 
video usage behavior. In addition, we expanded upon their questions given our specific research 
questions and interests. We added questions about the value of online homework, peer interaction, 
and the online discussion board; applicability to one’s future career; perceived learning gains and 
the development of computer programming skills; level of perceived responsibility on the student; 
and student motivation with the flipped classroom. A copy of our survey is shown in Appendix A. 
As with the CUCEI, we distributed the evaluation survey during the last week of class and collected 
the data anonymously. 

We also asked two open-ended questions on benefits as well as drawbacks and suggestions 
regarding the flipped classroom. A content analysis of each question was performed by a single 
coder, who was an upper-level engineering student. A second coder, the assessment analyst for the 
project, coded 50% of the responses to the benefits question to provide a measure of inter-rater 
reliability. The inter-rater reliability based on Cohen’s Kappa was k = 0.75, which suggests good 
agreement beyond chance (Norusis, 2005). The coding framework was developed using a grounded, 
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emergent qualitative analysis as part of prior flipped classroom research by the assessment analyst 
(Neuendorf, 2002; Clark et al., 2014; Clark et al., 2016). The drawbacks/suggestion question was 
analyzed in the same manner, with the assessment analyst coding 48% of the responses to provide 
a measure of inter-rater reliability. The inter-rater reliability achieved for this question was k = 0.76, 
showing strong agreement beyond chance (Norusis, 2005). 

We also sought student perceptions and feedback using focus groups in the fully-flipped course. 
Focus groups provide a means to obtain qualitative information from a group of people on their 
experiences and perceptions, which can be used alongside survey data for triangulation purposes 
(Fitzpatrick, Sanders, & Worthen, 2011). We conducted two focus groups at approximately the 
three-fourths point in the semester, each consisting of a different student demographic. One of the 
groups consisted of white males, and the other group consisted of students other than white males, 
including Hispanic and female students. This was consistent with our interest in investigating these 
instructional methods for underrepresented minorities and females in engineering. In the discussion 
of our results, we discuss our survey and focus group data together to triangulate our findings. In 
the focus groups, we asked questions about the perceived benefits and disadvantages of the flipped 
classroom, including learning or professional growth, challenges or negative outcomes, individual¬ 
ized support, and impact on programming skills. 

Since our focus group questions aligned with the open-ended questions from our flipped classroom 
evaluation survey, we used the coding framework for the open-ended questions to analyze the focus 
group responses in a structured way. The same two coders (i.e., the upper-level engineering student 
and the assessment analyst) coded the focus group data using the framework. Both components of 
the overall framework were used - the benefits as well as the drawbacks/suggestions. Even though 
the responses were double-coded, we calculated a first time reliability based on Cohen's kappa. 
The first time inter-rater reliability for use of the benefits framework with the focus group data was 
k = 0.79, showing strong initial agreement. For the drawbacks/suggestions framework, k = 0.69, 
showing fair initial agreement. 


RESULTS 


In this section, we provide a comparison of the final exam results for the three methods of in¬ 
struction for various demographic segments to address our first research question. These final exam 
results are based on both multiple-choice and free response questions. Additionally, flipped versus 
blended results are provided for specific topic areas within numerical methods, such as integra¬ 
tion and differentiation. We also provide results from the various student perception surveys (i.e., 
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classroom environment and flipped-classroom evaluation) as well as the student focus groups to 
address our second and third research questions. 

Direct Assessment of Learning 

Comparison of Methods: Multiple-Choice Questions 

We compared the multiple-choice results for the three instructional methods for all students as 
well as for various demographic segments of interest, as shown in Table 3. The p-values based upon 
the parametric and non-parametric analyses of covariance were generally in agreement. Given this 
and our tendency to default to the non-parametric analyses in the presence of the small sample 
sizes, only the non-parametric (i.e., Quade’s Test) results are shown. Also shown are the adjusted 
mean scores (out of a possible 14 points), the p-values for the individual-level tests (i.e., prior to 
adjustment using Bonferroni’s correction), Tamhane’s post-hoc comparisons, and the sample sizes. 
Given the large number of possible effect sizes and limited table space, only effect sizes for those 
pairs with the smallest p-values are shown in Table 3. A list of all large effect sizes for the multiple- 
choice results can be found in Table 4. 

As shown in Table 3, the flipped or semi-flipped method was always associated with a higher 
adjusted mean relative to the blended method. Given the 15 tests shown in Table 3, the a-level for 
each individual test was reset to 0.05/15 = 0.003. Although none of the individual tests would be 
considered significant upon applying this strict criterion, we did uncover several large effect sizes 
as shown in Tables 3 and 4, pointing to practical significance. For all students considered together, 
the effect size was medium (d=0.54) when comparing flipped versus blended instruction. For males, 
the semi-flipped adjusted mean was higher than the blended mean, and the effect size was large 
(c/=0.90). For community college transfers with an Associate’s degree, there was some evidence 
that the semi-flipped was better than the blended approach, in particular given the large effect 
size (c/=1.09). For medium effort students (40-65 hours/week), both the flipped and semi-flipped 
methods had higher means than the blended approach, and the effect sizes were large. Non-minority 
students scored higher with semi-flipped versus blended instruction, and the effect size was medium 
(d= 0.76). The small sample sizes associated with semi-flipped approach potentially influenced the 
levels of significance. 

Additional large effect sizes pointing to practical significance were found for several pairs of 
methods. These additional pairs are provided in Table 4, along with those from Table 3. The reader 
will want to keep in mind that some demographic segments had small sample sizes in the semi- 
flipped classroom. However, notable are the results for “other transfer” students and medium-effort 
students, in which the effect sizes were large when comparing the flipped to the blended classrooms 
(c/=1.37 and d= 0.85, respectively). Recall that “other transfers” consisted of internal-to-USF transfers 
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Multiple-Choice 
(14 points) 

Flip 

F 

Semi-F 

SF 

Blended 

B 

Quade’s Test 

Cohen 

Effect 

Size 

d 

F 

SF 

B 

Overall 

P 

Sig 

Pair 

Post 
Hoc p 


Sample 

Size 

Adjusted Mean 

All 

9.57 

9.86 

8.37 

0.016 

F&B 

0.053 

0.54 

41 

18 

73 

Gender 











Male 

9.57 

10.32 

8.34 

0.008 

SF&B 

0.005 

0.90 

34 

15 

64 

Female 

9.60 

6.99 

8.65 

0.417 




7 

2 

9 

Age 











<= 22 

9.95 

10.20 

8.85 

0.075 




25 

12 

41 

>22 

8.95 

8.53 

7.74 

0.402 




16 

4 

32 

Transfer 











Fresh Adm. 

9.83 

10.37 

9.26 

0.551 




15 

7 

34 

CC Assoc. 

9.00 

9.88 

7.49 

0.015 

SF&B 

0.074 

1.09 

18 

9 

33 

Other 

10.32 

7.96 

8.25 

0.048 




8 

1 

6 

Effort 











<40 

9.54 

10.02 

8.91 

0.839 




12 

5 

17 






F&B 

0.013 

0.85 




40-65 

9.59 

9.76 

7.87 

0.003 




21 

10 

44 






SF&B 

0.023 

0.93 




>65 

9.35 

11.06 

9.37 

0.459 




7 

2 

12 

URM 











Yes 

9.32 

8.66 

7.87 

0.232 




19 

4 

13 

No 

9.82 

10.18 

8.47 

0.027 

SF&B 

0.041 

0.76 

22 

14 

60 

Pell Grant 











Yes 

9.69 

9.87 

8.40 

0.152 




12 

7 

29 

No 

9.54 

9.95 

8.36 

0.108 




29 

10 

44 


Note: The /)-values shown are those prior to application of Bonferroni’s correction. 

Table 3. Multiple-Choice Questions - Comparison of Methods. 


to the engineering school, community college transfers without Associates degrees, and transfers 
from external four-year programs. Another type of transfer student, community college transfers 
with Associates degrees, demonstrated greater achievement with the semi-flipped versus blended 
classroom (c/=1.09), as did medium-effort students and males. These results, along with the adjusted 
means shown in Table 3, lead us to preliminarily conclude that some degree of flipped instruction 
may have been more beneficial than blended instruction for lower-order skills development in our 
numerical methods course. 
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Demographic 

Mean 1 

Mean 2 

Cohen’s d 

Other Transfer 

F 

SF 

1.56 

Other Transfer 

F 

B 

1.37 

Female 

F 

SF 

1.18 

CC Transfer w/ Assoc. 

SF 

B 

1.09 

Effort 40-65 

SF 

B 

0.93 

Male 

SF 

B 

0.90 

Effort 40-65 

F 

B 

0.85 

Effort > 65 

SF 

F 

0.84 

Effort > 65 

SF 

B 

0.83 


Note: Mean 1 > Mean 2 

Table 4. Large Effect Sizes (Multiple-Choice Questions). 


Comparison of Methods: Free Response Questions 

Although we anticipated that flipped instruction would emerge as the best method for the free 
response questions, as the students had to “dig deeper” on their own, we did not find statistically 
significant differences in the three methods for any demographic group. The results are provided in 
Table 5, in which the majority of the demographic segments scored highest with the semi-flipped 
approach (although not significantly so). For all students considered together, the semi-flipped 
approach was associated with the highest adjusted mean for the free response results; however, 
the associated effect sizes for the semi-flipped method compared to the other methods were small 
(C/- 0 . 30 ). 

However, we did find several large effect sizes related to higher-order skills development in 
numerical methods, as shown in Table 6. Notable are the results for “other transfer” students, in 
which the effect size was large for flipped versus blended instruction (c/=1.11). The under-represented 
minority (URM) students performed best with semi-flipped instruction, in which the effect sizes in 
comparison to flipped and blended instruction were large. Thus, based on Table 6, we have some 
initial data to suggest that flipped instruction (of some amount) may have been associated with 
enhanced higher-order skills attainment in our numerical methods course. 

Comparison of Methods by CPA Groups 

We also compared the instructional methods for students of different pre-course achievement 
backgrounds, as determined by the pre-requisite GPA. Students in the highest GPA group (i.e., 3.50 
and above) scored higher with semi-flipped vs. blended instruction on the multiple choice questions, 
as shown in Table 7. The effect size was large (c/=1.27), although the result was not significant after 
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Flip Semi-F Blended Quade’s Test F SF B 

F SF B - - 

Free Response ___ Overall Sig Post Sample 

(16 points) Adjusted Mean p Pair Hoc p Size 


All 

5.92 

6.85 

5.98 

0.693 

41 

18 

73 

Gender 








Male 

5.99 

7.14 

5.89 

0.458 

34 

15 

64 

Female 

5.57 

5.00 

6.78 

0.342 

7 

2 

9 

Age 








<= 22 

6.48 

6.95 

6.62 

0.966 

25 

12 

41 

>22 

5.11 

5.56 

5.12 

0.871 

16 

4 

32 

Transfer 








Fresh Adm. 

7.13 

7.01 

7.03 

0.898 

15 

7 

34 

CC Assoc. 

4.21 

6.82 

5.09 

0.142 

18 

9 

33 

Other 

7.63 

4.91 

5.34 

0.099 

8 

1 

6 

Effort 








<40 

4.63 

8.22 

6.26 

0.197 

12 

5 

17 

40-65 

6.45 

6.64 

5.78 

0.624 

21 

10 

44 

>65 

6.61 

6.17 

6.20 

0.887 

7 

2 

12 

URM 








Yes 

5.64 

8.54 

5.83 

0.391 

19 

4 

13 

No 

6.22 

6.38 

6.00 

0.897 

22 

14 

60 

Pell Grant 








Yes 

4.94 

5.72 

5.95 

0.627 

12 

7 

29 

No 

6.34 

7.64 

6.04 

0.592 

29 

10 

44 


Note: The /?-values shown are those prior to application of Bonferroni's correction. 


Table 5. Free Response Questions - Comparison of Methods. 


Demographic 

Mean 1 

Mean 2 

Cohen’s d 

Other Transfer 

F 

SF 

1.31 

Other Transfer 

F 

B 

1.11 

URM - Yes 

SF 

F 

1.04 

Effort < 40 

SF 

F 

1.00 

URM - Yes 

SF 

B 

0.97 

Female 

B 

SF 

0.97 

CC Transfer w/ Assoc. 

SF 

F 

0.95 


Note: Mean 1 > Mean 2 

Table 6. Large Effect Sizes (Free Response Questions). 
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Flip 

F 

Serni-F 

SF 

Blended 

B 

ANOVA 

P 

K-W 

P 

Sig 

Pair 

Post 
Hoc p 

Cohen 
Effect 
Size d 

F 

SF 

B 



Mean 


Sample Size 

Multiple-Choice 












Prerequisite GPA 











Under 3.00 

9.18 

8.71 

7.61 

0.111 

0.134 




11 

7 

28 

3.00 to 3.49 

9.30 

9.00 

8.27 

0.368 

0.630 




20 

4 

26 

3.50 + 

10.70 

11.86 

9.42 

0.019 

0.033 

SF&B 

0.041 

1.27 

10 

7 

19 

Free Response 












Prerequisite GPA 











Under 3.00 

5.09 

4.71 

5.11 

0.939 

0.922 




11 

7 

28 

3.00 to 3.49 

6.05 

5.50 

5.39 

0.757 

0.664 




20 

4 

26 

3.50 + 

6.80 

10.29 

7.79 

0.069 

0.091 




10 

7 

19 


Note: The p-values shown are those prior to application of Bonferroni’s correction. 


Table 7. GPA Groups - Comparison of Methods. 


applying the Bonferroni correction. The free response score for this top GPA group was also highest 
with the semi-flipped approach, and the associated effect sizes were large, with c/=1.17 and d= 0.84 
when comparing to the flipped and blended methods, respectively. For the other GPA groups, the 
effect sizes were medium or small, with a maximum absolute value of d= 0.72. 

Comparison of Methods by Topic Area 

To investigate whether certain instructional methods might be better for certain numerical 
methods topic areas, we compared the results of the multiple-choice questions associated with 
each of the seven topic areas in the course, using the prerequisite GPA as the control variable. We 
used data from the two semesters in which fully-flipped and blended instruction were used, as 
shown in Table 8. Based on an ANCOVA, there were four topic areas for which the flipped method 
resulted in higher scores. The most notable difference was for the introductory scientific comput¬ 
ing area, which had a medium effect size of d= 0.73. Given the highly significant result associated 
with scientific computing (p<0.0005), the difference is significant even after applying Bonferroni’s 
correction (0.05/7=0.007). This topic area covers motivations for using numerical methods, their 
applications, and identification and handling of error. A substantial difference occurred for nonlinear 
equations, which had higher scores with flipped instruction and a medium effect size. Differences 
for differentiation and simultaneous linear equations are also notable. The other topic areas had 
small effect sizes. Each topic area was associated with two questions worth one point each. Based 
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Topic Area 
(2 points each) 

Flip 

F 

Blended 

B 

ANCOVA 

P 

Cohen 

Effect 

Size 

d 

F 

B 

Adjusted 

Mean 

Sample 

Size 

Differentiation 

1.55 

1.27 

0.026 

0.44 

41 

73 

Integration 

1.41 

1.50 

0.427 

0.15 

41 

73 

Interpolation 

1.31 

1.41 

0.425 

0.16 

41 

73 

Scientific Computing 

1.43 

0.93 

<0.0005 

0.73 

41 

73 

Nonlinear Eq. 

1.16 

0.81 

0.009 

0.51 

41 

73 

Ordinary Differential Eq. 

1.51 

1.46 

0.722 

-0.07 

41 

73 

Simultaneous Linear Eq. 

1.19 

0.96 

0.059 

0.37 

41 

73 


Note: The ^-values shown are those prior to application of Bonferroni’s 
correction. 

Table 8. Topics Areas - Comparison of Flipped vs. Blended. 


on these results, flipped instruction may have been the better approach for several topic areas when 
considering the students as a whole. 

Student Perceptions and Preferences 
Classroom Environment inventory 

To assess the psychosocial dimensions of our three classroom environments and investigate 
our second research question about perceptions of the learning environments, we used Fraser’s 
College and University Classroom Environment Inventory (CUCEI) as shown in Table 9. We 
obtained response rates in the range of 67% to 78% of the class enrollments across the three 
classrooms. Upon applying Bonferroni’s correction, there were no significant differences with 
the classroom environment dimensions. Task orientation, which was associated with the small¬ 
est p-value, was rated higher in the blended and semi-flipped classrooms versus the flipped 
classroom. This perception of less organization in the flipped classroom makes sense, as stu¬ 
dents were expected to conduct their own problem solving efforts with the instructor present 
to guide. Thus, in general, the classroom environments were statistically equivalent across the 
three methods of instruction. All Cohen’s d effect sizes were less than 0.50 in absolute value, 
indicating small effects. 

Although not statistically significant, the following trends were noticed. The flipped mode was 
rated lower than the blended mode on six of the seven dimensions. The semi-flipped mode was rated 
lower than the blended mode on four of the seven dimensions. This leads to the following ques¬ 
tion: Do students prefer the environment of the blended classroom to the flipped (or semi-flipped) 
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Semi-F Blended _ , 

Flip F SF B _ , ^ f hel 

. Post Effec 

(n=43) (n=18) (n=74) A nOVA K -W Sig Hoe Size 


Dimension Mean p p Pair p d 


Cohesiveness 

Students know & help 
one another 

2.80 

2.93 

3.07 

0.19 

0.18 

Individualization 

Treated individually or 
differentially 

2.61 

2.51 

2.45 

0.22 

0.39 

Innovation 

Novel class activities or 
techniques 

2.92 

2.99 

3.09 

0.23 

0.19 

Involvement 

Active participation in 
class 

3.29 

3.22 

3.31 

0.87 

0.66 

Personalization 

Interaction w/ instructor 

4.06 

3.90 

4.13 

0.41 

0.21 

Satisfaction 

Enjoyment of classes 

3.51 

3.73 

3.61 

0.65 

0.69 

Task Orientation 

Organization of class 

4.08 

4.32 

4.31 

0.045 

0.055 B&F 0.068 0.47 


activities 


Note: The p-values shown are those prior to application of Bonferroni’s correction. 

Table 9. Classroom Environment - Comparison of Methods. 


classroom for numerical methods? The further collection of data will enable us to answer this ques¬ 
tion with more certainty. The small sample size in the semi-flipped section likely influenced the 
significance levels and contributes to the preliminary nature of these results. As might be expected, 
individualization was rated higher in the flipped classroom versus in the other classrooms, although 
not significantly so. This dimension measures individual or differential treatment, which is possible 
with in-class problem solving efforts. 

Flipped Classroom Evaluation Survey 

The students evaluated the flipped portion of the summer course as well as the fully-flipped 
course in the fall via a flipped classroom evaluation survey, with approximately 68% of enrolled 
students responding. This, in conjunction with the student focus groups, enabled the investigation 
of our third research question about the perceived benefits and drawbacks of flipped instruction. 
Thirty-eight percent (38%) of the respondents preferred the flipped classroom to usual methods of 
instruction. Another 38% were unsure of their preferences, and 24% did not prefer flipped instruc¬ 
tion. In a recent school-wide initiative, a similar pattern involving an approximate three-way split 
in student preference for the flipped classroom was found, with 27% indicating a preference, 36% 
indicating a non-preference, and 36% being unsure (Clark et. al., 2016). 
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However, when asked to compare the use of class time for problem solving with the instructor 
present versus listening to a lecture, 54% of the numerical methods respondents preferred the 
former. In comparison, Zappe et al. found slightly lower student preference for active learning in 
the classroom, with 48% agreeing or strongly agreeing that they preferred problem solving versus 
lecture during class time (Zappe et al., 2009). This pattern of a lower preference for the flipped 
classroom overall compared to a higher preference for active learning during class has been noted 
previously (Bishop & Verieger, 2013). These researchers offered the explanation that students tend 
to prefer in-person lectures to video lectures but ultimately prefer classroom activities to lecture. 

In terms of required effort, over 77% of respondents reported that this flipped classroom required 
more or much more effort compared to usual methods of instruction, and 84% felt it placed more or 
much more responsibility on them. In his post-course interview, the instructor also noted that stu¬ 
dents were given (and assumed) more responsibility for their own learning with the flipped format; 
as discussed in the introduction, this was one of his goals with flipping the course. 

The respondents felt the online, auto-graded, multiple-attempt homework problems were valu¬ 
able (79% agreement or strong agreement), as was the discussion board (74% agreement or strong 
agreement). In terms of other aspects of the flipped classroom, 36% agreed or strongly agreed 
that the learning gains were better in the flipped classroom. Forty-four percent (44%) agreed or 
strongly agreed that interaction with their peers in the flipped classroom was valuable. Thirty-four 
percent (34%) thought the flipped classroom led to valuable experiences for their future careers. 
Finally, only 25% felt the flipped classroom enabled them to develop better computer programs for 
numerical methods solutions. 

We asked the students who received some degree of flipped instruction to report the percent¬ 
age of videos on the assigned topics they watched. The average percentage reported was 62%. 
However, the videos were not the only available resource for learning the material; students were 
also directed to the book chapters, which were online as well as in hardcopy format. In fact, 79% of 
survey respondents reported using a combination of the videos and the book chapters. Only one 
student reported using just the videos. In a post-course interview, the instructor felt that students 
took responsibility for the self-learning aspect of the flipped classroom during both the summer and 
fall terms. In a first-year engineering course that used pre-class videos, the students who reported 
having prepared for class also used a combination of readings and videos. The authors concluded that 
providing students with a variety of options for their preparation was desirable (Hamlin et al., 2014). 

Content Analysis of Benefits 

In an open-ended question on the flipped classroom evaluation survey, we asked the students what 
they liked about the flipped classroom and the benefits they perceived. The frequencies associated 
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Frequency 

% of 

Respondents 

Category 

Description 

25 

43% 

Enhanced Learning 
or Learning Process 

Better understanding; less confusion 

Enhanced leaming/effectiveness/depth/ability 
Subject matter retention 

Multiple sources/resources for understanding 
Reinforcement and review 

Multiple attempts 

20 

34% 

Preparation, 
Engagement 
& Professional 
Behaviors 

Engaged during class; paid attention; not bored 
Enjoyed class 

Arrived to class prepared 

Ability to leam on one’s own 

Drove motivation and accountability 

13 

22% 

Video/Online 

Learning 

Re-watch videos 

Work at one’s own pace; pause video 

Flexibility, convenience, own preferences 
Modularization of topics 

10 

17% 

No Benefit or 
Neutral Result 

No benefits perceived 

Did not like flipped instruction 

Videos not used 

Instructional differences not noticed 

8 

14% 

Alternative Use of 
Class Time 

In-class active learning, problem solving, clickers 
In-class support and questions 

In-class group time for projects 

Student interactivity and peer support 

4 

7% 

Specific to Course 
or Course’s Videos 

Videos concise 

Videos had a good pace 

Overall work time less 

Videos had relevant content (e.g., demo or 
examples) or were of high quality 


Table 10. Summary of Open Ended Responses to Benefits. 


with the categories in our coding framework are shown in Table 10. To our satisfaction, the most 
frequently perceived benefit was enhanced learning or learning processes, as perceived by 43% of 
respondents. This was similar to the result from our closed-ended question about learning gains, 
in which 36% agreed or strongly agreed that the learning gains were better with the flipped class¬ 
room. In addition, thirty-four percent (34%) identified preparation, engagement, and professional 
behaviors as a benefit. In his post-course interview, the instructor identified life-long learning skills 
as a benefit to the students. His flipped classroom aimed to prepare students to be able to learn 
on their own for when they are working for a company someday. In addition, he wanted to prepare 
students to independently learn using multiple sources, such as books, videos, or other online con¬ 
tent, as today's world is replete with a multitude of informational formats. In summary, one-third 
to almost one-half of the respondents perceived the top benefits that we were hoping to achieve 
with the flipped classroom. These results were based on a content analysis of 58 student responses. 
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Comparison with Focus Group Results 

In our focus groups, enhanced learning and learning processes, the alternative use of class time, 
and engagement and professional behaviors were the three benefits identified most frequently by 
each of the demographic groups (i.e., white males as well as underrepresented minority and female 
students). This aligned fairly closely with the results of the open ended question about the benefits. 
The white males indicated that the flipped classroom led to deeper learning because of the need to 
explore material on their own. Also, one could spend as much time as needed to understand some¬ 
thing. In the focus group with students other than white males, better understanding and a better 
idea of mechanical engineering, including everyday scenarios, were stated as benefits of the flipped 
classroom. For both demographics, reinforcement and review of lecture material and a variety of 
resources created enhanced learning processes. Both demographics also highlighted the alterna¬ 
tive use of class time in the form of the ability to ask questions and receive support during class as 
benefits. For both demographics, the third most frequently identified benefit was the connection 
between the flipped classroom and their future professional careers as well as their engagement 
and preparation for class. The white males in particular stated that the flipped classroom was good 
professionally because it encouraged responsibility for one’s own learning. The underrepresented 
minority and female students explained that the videos will be a valuable resource in the future, as it 
is difficult to keep in mind every possible numerical method that could be applied. Also, the flipped 
classroom promoted motivation, in-class engagement, and preparation with both demographics. 
When comparing the focus group results between the two demographic groups using the coding 
framework (i.e., Table 10), there was very little difference in the responses. 

Content Analysis of Suggestions and Drawbacks 

In a second open-ended question on the evaluation survey, we asked the students what draw¬ 
backs they perceived with the flipped classroom and suggestions for improvement. The frequencies 
associated with the categories in our coding framework are shown in Table 11. The most frequent 
suggestions or drawbacks pertained to the use of in-class time, as identified by 43% of respondents. 
This was followed by the perceived load, burden, or stressors on the students (34% of respondents), 
such as an increased time burden. Eleven percent (11%) of the respondents suggested better prepar¬ 
ing and equipping students for flipped instruction. The results in Table 11 were based on a content 
analysis of 56 student responses. 

Comparison with Focus Group Results 

In our focus groups, the three drawbacks/suggestions identified most frequently were related to 
the use of class time, workload and stressors, and preparation for the flipped style of instruction. 
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Frequency 

% of 

Respondents 

Category 

Description 

24 

43% 

In-Class Time 

Increase time for active learning or problem solving 

Increase effectiveness or relevancy of problems; grade them 
Provide appropriate amount of lecture or content review 

Have more instructor-types during class to assist 

Synchronize class activity and video content 

19 

34% 

Load, Burden, Stressors 

Insufficient time to complete out-of-class activities 

Increased work load 

Increased time burden 

Concerns over grades or impacts to the grade 

Accountability quizzes (including surprise) 

10 

18% 

Specific to Course or 
Course’s Videos 

Include more examples or problems in the videos 

Videos needed editing or bug/technical fixes 

Videos were too long 

Videos were not sufficiently described 

Videos were dry or boring 

Videos did not have an appropriate pace 

Videos repeated information 

Video material was too complex 

7 

13% 

No Drawbacks or Neutral 
Result 

No drawbacks or suggestions 

6 

11% 

Approach Differently 

Do not flip courses in general; use traditional teaching 

Do not flip this course in particular 

Provide students with a choice on flipping 

Flip only a portion of the class periods 

6 

11% 

Prepare, Equip & Incentive 
Students To Flip 

Prepare students for the flipped learning style 

Incentivize students, including video quizzes 

Clarify/emphasize expectations, including video watching 

Provide video “lecture” notes 

Ensure videos available in advance for students 

5 

9% 

Student Learning 

Lesser understanding or learning 

Difficulty learning from a video 

2 

4% 

Video/Online Learning 

Students unable to ask questions during a video 

Instructor unable to sense student understanding in a video 
Distractors to viewing videos in a non-classroom setting 

Less motivation to attend class 


Table 11. Summary of Open Ended Responses to Suggestions/Drawbacks. 


This aligned closely with the results of the open-ended question, as shown in Table 11. The non-white 
males identified suggestions for the improved use of class time more frequently than the white 
males did, including more or alternative in-class activities and problems as well as additional in-class 
review of questions and concepts related to the clickers or MATLAB. Specifically, the non-white 
males suggested that more problems and examples be provided during class, including more dif¬ 
ficult and in-depth problems. Another request or suggestion was to do more active learning during 
class in the form of MATLAB programming, using laptops if necessary. According to these students, 
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this would drive confidence and lessen their difficulties with programming, because the instructor 
would be present to answer questions and assist. The white males noted to a larger extent the load 
or stressors they felt with the course, most notably increased work or time. Both demographics 
equally emphasized the need to prepare and equip students for the flipped classroom. This included 
suggestions that the instructor fully explain at the start of semester that the course is being flipped 
and that he/she clarify exam expectations in the new class format. 


DISCUSSION OF RESULTS 


Considering our first research question, the flipped or semi-flipped method was typically associated 
with a higher adjusted mean versus the blended method for either the multiple-choice or free-response 
exam questions. In addition, although none of the tests was significant upon applying Bonferroni’s 
correction, we found several large effect sizes for flipped or semi-flipped versus blended instruction, 
leading us to preliminarily conclude that some degree of flipped instruction may have been better than 
blended learning for achievement in the course. Specifically, for all students considered together, the 
effect size was medium (c/=0.54) for flipped versus blended instruction when examining lower-order 
skills attainment. There were some promising trends for transfer students to the college of engineer¬ 
ing. For those students who transferred with a community college Associate’s degree, we found some 
evidence that semi-flipped instruction may have been better than blended instruction, given the large 
effect size (c/=1.09). For “other” transfer students, we found large effect sizes for flipped versus blended 
instruction for both lower-order and higher-order skills achievement (d= 1.37 and d= 1.11, respectively). 
Although our results do not point to significant differences at this point for URM students, the effect 
sizes were large for semi-flipped instruction versus the other two methods for the higher-order skills. 

When comparing these methods for students of varying pre-course achievement backgrounds, 
we found a large effect size for the highest GPA group and the multiple-choice questions, in which 
semi-flipped exceeded blended performance. Finally, we found a significantly-higher score for flipped 
versus blended instruction for the scientific computing topic area (p<0.0005), with a medium effect 
size (d= 0.73). We also found a substantial difference and a medium effect size for nonlinear equa¬ 
tions, with flipped exceeding blended instruction. Based on these various analyses, we conclude in 
relation to our first research question that flipped instruction may have led to greater achievement 
for certain numerical methods topic areas versus a purely blended approach. 

Although exam results seemed to favor some degree of flipped instruction, the classroom en¬ 
vironment results may have favored the blended approach, coinciding with our second research 
question. Although the classroom environment was actually statistically equivalent across the three 
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methods, with small effect sizes, the flipped classroom was rated lower than the blended classroom 
on six of the seven dimensions, and the semi-flipped was rated lower than the blended mode on 
four dimensions. 

Regarding the third research question, our students tended to view the flipped classroom as de¬ 
manding, with large percentages reporting increased effort and responsibility required on their part. 
One possible explanation for the perceived greater effort is the inability for students to be passive 
during class. Thus, although the flipped classroom may not require more time overall, it does require 
students to be active during class. Although just 38% preferred the flipped classroom, 54% stated 
a preference for solving problems in class versus listening to a lecture. The most frequently-stated 
benefits of flipped instruction, as determined through a content analysis of an open-ended ques¬ 
tion, were enhanced learning or learning processes (43% of respondents); preparedness, engage¬ 
ment, and professional behaviors (34%); and conveniences afforded by video learning (22%). This 
was corroborated by the focus group results, in which the three benefits identified most frequently 
were enhanced learning or learning processes, the alternative use of class time, and engagement 
and professional behaviors. When asked about the benefits of flipped instruction in a post-course 
interview, the instructor also corroborated these findings, identifying life-long learning, career 
preparation, and enhanced responsibility for learning. 

Study Limitations 

Our study is a quasi-experimental design, as are many educational intervention studies, given 
the non-random assignment of students to the three classrooms that were compared. However, 
to control for a student’s previous academic performance, a likely confounding factor, we used 
the pre-requisite GPA as a control variable in our analysis of their exam scores. Furthermore, we 
believe that more data is needed to fully investigate the research questions, especially given the 
large number of demographic tests and Bonferroni’s correction. The sample sizes associated with 
some of the demographics segments are small, impacting significance levels and the ability to draw 
conclusions. This additional data will be collected over the remainder of our research grant. We did, 
however, use conservative statistical procedures (i.e., non-parametric tests and effect sizes) in the 
presence of these small samples. 

Related to this, we acknowledge that our study was done with engineering students from one 
university. Therefore, these students may not be a representative sample of all engineering students 
who take numerical methods, thereby limiting the generalizability and applicability of the results to 
other students at other schools. In addition, this study involved just one instructor, albeit a seasoned 
numerical methods instructor. Although a single instructor allowed for more controlled compari¬ 
sons among the methods, it is possible that our results would be different if another instructor had 
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developed and taught the course. To this end, our grant involves two additional universities. Once 
these two additional instructors complete their flipped implementations using the same methods 
as described here, we will be able to add to our findings and report results of students from three 
schools and three engineering disciplines. Thus, instructors will want to keep in mind that we con¬ 
sider our results preliminary at this point; however, the trends we have demonstrated will contribute 
to our overall findings at the completion of the grant study. 

Future Research 

As discussed in the introduction, enhancement of metacognitive skills was one of the instruc¬ 
tor’s main goals in flipping the course, along with enhanced responsibility and higher-order skills 
development. Although we did not directly assess students’ metacognitive skills, this may be a 
desirable future research line. In his post-course interview, the instructor discussed that the open- 
ended responses submitted by the students before class, which contained points of difficulty or 
interest, were of much higher quality and complexity in the flipped versus blended classrooms. He 
also explained that students were required to re-do and reflect on exam questions they did not get 
correct during each of the three semesters. Each student had to provide a small essay explaining 
why he/she got the problem wrong and why his/her re-submission was correct. The instructor was 
very pleased with the students’ reflections, stating that the submissions were of extremely good 
and similar quality across the three semesters. Some students even pointed out they were not pre¬ 
pared in some cases for the exam questions. Since reflection is a valuable component of engineering 
practice, we suggest future research to determine whether there are measurable differences among 
these instructional methods in regards to student metacognition. 

In addition, despite greater effort and responsibility perceived during the course by students, 
they still saw longer-term benefits with flipped instruction, including enhanced learning processes 
and career preparation. Therefore, should we be assessing the impacts of flipped instruction further 
into the future with our students? This is a great research opportunity, and we may be pleased to 
learn the outcomes. Related to this, instructors and researchers may wish to identify and consider 
additional direct assessment outcome variables (besides exam scores) to better demonstrate sig¬ 
nificant differences or improvements with the flipped classroom. 


CONCLUSIONS 


Blended, flipped, and semi-flipped approaches to teaching a mechanical engineering course in 
numerical methods were taken at the University of South Florida beginning in the spring of 2014. This 
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enabled a comparison of these modalities in terms of student performance on both multiple-choice 
and free response exam questions and classroom environment perceptions. This paper is believed to 
be one of the first such comparisons within engineering education. The instructor has been teaching 
this course in a fully-guided, blended fashion for approximately 20 semesters and transitioned to 
a fully-flipped format in the fall 2014. With the flipped instruction, students applied and practiced 
concepts and skills initially obtained outside the classroom. Blended instruction involved maintaining 
the traditional in-class lecture enhanced by online and technology-based resources, including clickers, 
online quizzes, a discussion board, and a library of videos. The semi-flipped approach incorporated 
both of these instructional methods during the semester in an approximately-equal split of the time. 

Based on our exam results, we believe that flipped instruction (or some degree of it) may have 
been the preferred method for achievement in our numerical methods course, relative to blended 
instruction. The instructor’s preferred format for teaching this course is the semi-flipped approach, 
which he believes allows more time to guide students through difficult problems and is less im¬ 
pacted by larger class sizes. The classroom environment, however, was rated highest in the blended 
classroom, although not significantly so. Students perceived both benefits and drawbacks with 
flipped instruction. Although the needed effort and responsibility levels were perceived as higher 
in the flipped classroom, longer-term benefits were realized as well, including career preparation, 
professional behaviors, and enhanced learning or learning processes. The students tended to have 
suggestions for the use of class time in the flipped environment, including more hands-on practice 
with MATLAB programming or exposure to more challenging problems. As we continue to study 
these important research questions, we anticipate adding to our understanding of blended versus 
flipped instruction as regards achievement and preferences in numerical methods, in particular for 
non-traditional and under-represented students. 
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APPENDIX A 


Flipped Classroom Evaluation Survey (modeled upon Zappe et al., 2009) 



Question 

Response Options or Type 

1 

Do you prefer a flipped classroom 
over the usual method of instruction in 
this class? 

O Yes 

O No 

O Not sure yet 

2 

Please indicate the method that best describes 
how you completed your out-of-class learning. 

O I only viewed the videos. 

O I only read the book chapters. 

O I used a combination of the videos and the book chapters. 

O I used other resources. 

O I did NOT use the videos, book chapters, or any other resources for 
out-of-class learning. 

3 

What percentage of the videos on the assigned 
topics did you watch? (Approximate as 
needed, and use 0 or 100 as appropriate.) 

0-100% 

4 

When did you primarily view the videos? 

O Before the class period for which they were assigned 

O After the class period for which they were assigned 

5 

How often did you re-watch the videos or any 
portions of them? 

O Never 

O Rarely 

O Sometimes 

O Often 

O Almost always or always 

6 

Why did you re-watch videos or portions of 
them? (Select all that apply) 

O The topic was difficult or challenging to grasp. 

O The instructor’s explanation in the video was not clear. (Please 
provide specifics) 

O To reinforce my understanding as I was learning new material. 

O To review or study course material prior to an exam or homework 
problem. 

O Poor audio or visual quality of the video or other technical difficulty. 

(Please provide specifics) 

O Other (Please provide specifics) 

7 

With the flipped classroom, how would 
you rate the overall effort required of you, 
compared to the usual method of instruction in 
this class? 

O Much less 

O Less 

O About the same 

O More 

O Much more 

8 

How would you rate the overall effort required 
of you in this class compared to other college/ 
university engineering classes (either flipped or 
non-flipped) that you’ve taken or are currently 
taking? 

O Much less 

O Less 

O About the same 

O More 

O Much more 

9 

I prefer using class time for hands-on activities 
or problem solving exercises (with the 
instructor or TAs present for assistance) rather 
than listening to a lecture. 

O Strongly disagree 

O Disagree 

O Neutral 

O Agree 

O Strongly agree 

10 

The online, automatically-graded, multiple- 
attempt homework problems were valuable to 
my learning in this course. 

O Strongly disagree 

O Disagree 

O Neutral 

O Agree 

O Strongly agree 
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Question 

Response Options or Type 



O Strongly disagree 


I often did NOT know how to begin solving 

O Disagree 

11 

the in-class problems assigned in the flipped 

O Neutral 


classroom. 

O Agree 



O Strongly agree 



O Much less 


With the flipped classroom, how would you 

O Less 

12 

rate the responsibility placed on you, compared 

O About the same 


to the usual method of instruction in this class? 

O More 



O Much more 



O Strongly disagree 



O Disagree 

13 

I am NOT able to learn from a video. 

O Neutral 



O Agree 



O Strongly agree 



O Strongly disagree 

14 

The flipped classroom enabled me to gain 

O Disagree 


valuable experience for my future career. 

O Agree 



O Strongly agree 



O Strongly disagree 


I had greater learning gains with the flipped 

O Disagree 

15 

classroom versus the usual method of 

O Neutral 


instruction in this class. 

O Agree 



O Strongly agree 



O Strongly disagree 

16 

More time needed to be spent during class 

O Disagree 

O Neutral 


reviewing the video or course content. 

O Agree 



O Strongly agree 



O Strongly disagree 


The ability to learn from and assist my fellow 

O Disagree 

17 

students in the flipped classroom was a 

O Neutral 


valuable learning outcome for me. 

O Agree 



O Strongly agree 



O Strongly disagree 


With the flipped classroom, I had the 

O Disagree 

18 

motivation to engage in the necessary learning 

O Neutral 


outside of the classroom. 

O Agree 



O Strongly agree 



O Strongly disagree 


I understand the reasons or rationale for the 

O Disagree 


flipped classroom style in this course. 

O Agree 



O Strongly agree 



O Strongly disagree 


The flipped classroom enabled me to develop 

O Disagree 

20 

better computer programs for numerical 

O Neutral 


methods problems. 

O Agree 



O Strongly agree 
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Question 

Response Options or Type 

21 

The course discussion board was a 
valuable component of my learning. 

O Strongly disagree 

O Disagree 

O Neutral 

O Agree 

O Strongly agree 

22 

What did you like most about the flipped 
classes and what benefits did you perceive? 

Open ended 

23 

What suggestions do you have for improving 
the flipped classes and what drawbacks did you 
perceive? 

Open ended 
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